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ABSTRACT 

A study examined 43 research projects conducted 
between FY89 and FY93 with funding under Section 353 of the Adult 
Education Act (AEA) grants and 4 additional research projects funded 
under other sections of the AEA. The project reports were collected 
by contacting each state literacy resource center, searching the ERIC 
system and a Division of Adult Education and Literacy guide to 
available 353 resources, and contacting individual contracting 
agencies directly. The purpose, methodology, and findings of each 
project for which a report was obtained were summarized in a brief, 
standard format. Similarities, correlations, and/or contradictions 
among the projects were identified. Most of the Section 353 projects 
were funded as demonstration projects delivering direct student 
services or staff development activities. Few projects had a true 
experimental design. The contact persons listed on each report were 
sent letters requesting information on follow-up activities and 
further developments since their projects' completion. Because only 
two responses were received, the follow-up portion of the study was 
abandoned. The project summaries were published in a 32-page booklet 
for distribution to interested adult educators through Pennsylvania's 
state literacy resource centers. (MN) 
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Abstract 



Title: 

Research Distilled: Practical Summaries of Adult Education 353 Research Projects, 1989-93 

Project No. 99-4024 Funding: $12,000 

Project Director: Tana Reiff Phone Number: (717) 299-8912 

Contact Person: Same 

Agency Address: 

New Educational Projects, Inc. 
P.O. Box 182 

Lancaster, PA 17608-0182 

Purpose: To provide, in an accessible format, information on research projects conducted 
under Section 353 grants. 

Procedures: Searched available lists of Section 353 projects conducted nationwide; discerned 
those projects of a research nature; obtained final reports of those research projects; reviewed 
and summarized the reports; compiled a 32-page booklet of these summaries. 

Summary of Findings: A total of 47 Adult Education Act projects were summarized: 43 con- 
ducted under Section 353 and four from other funding sources under the Act. 

Comments: An unanticipated finding was that severe gaps in the 353 dissemination system 
exist, as observed in the process of trying to secure reports from states other than Pennsylva- 
nia. Also, significant weaknesses in project design were discovered in all but a few of the 
reports. 

Product: A 32-page publication of research project summaries 
Descriptors: 
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Introduction 

Research Distilled: Practical Summaries of Adult Education 353 Research Projects, 1989-93 
was a staff development project funded under Section 353 of the Adult Education Act to pro- 
vide, in an accessible format, information on research projects conducted under Section 353 
grants. 

Toward that goal, Research Distilled pursued the following five objectives: 

1. Collect reports of research projects conducted under Section 353 special-projects grants 
in Pennsylvania and elsewhere in the United States since at least 1989. 

2. Summarize the purpose, methodology employed, and findings of each relevant research 
project, in a brief, standard format. 

3. Attempt to contact original researchers to learn what, if any, follow-up or subsequent 
developments have taken place since their projects were completed. 

4. Cross-reference related research projects to identify similarities, correlations, and/or con- 
tradictions. 

5. Produce 500 copies of a 32-page booklet of research-project summaries, for distribution 
through AdvancE to interested adult educators. 

These activities were conducted between July 1, 1993, and June 30, 1994. Project Director 
was Tana Reiff . Dr. Robert Zellers of the University of Pittsburgh, Johnstown, provided an 
academic perspective by reviewing all of the summaries written by Ms. Reiff prior to publica- 
tion. Cheryl Harmon of the AdvancE State Literacy Resource Center provided abundant, in- 
valuable assistance in retrieving documents related to the project. Sincere gratitude is ex- 
tended to her. In addition, several experts in adult education provided informal assistance, 
critiquing, and other advice during the course of the project. 

The audience for this report and the publication produced by this project would be adult 
education practitioners and researchers involved in either a preliminary literature search or an 
overview of research conducted under Section 353 between 1989 and 1993. 

Copies of this report as well as the project's publication, entitled Research Distilled: 
Summaries of Adult Education Act Section 353 Research Projects, 1989-93, are filed with the 
Pennsylvania State Literacy Resource Centers, Pennsylvania Department of Education. Cop- 
ies of the publication are available to keep while the supply lasts. The addresses of the re- 
source centers are as follows: 

AdvancE 

Pennsylvania Department of Education Resource Center 
333 Market Street, 11th Floor 
Harrisburg, PA 17126-0333 
(717) 783-9192 
1-800-992-2283 (in PA) 

Western Pennsylvania Adult Literacy Resource Center 
5347 William Flynn Highway (Route 8) 
Gibsonia, PA 15044 
1-800-446-5607, ext. 216 (in PA) 
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The Project 

The Research Distilled project was developed on the premise that no compilation of summa- 
ries of research projects conducted under Section 353 exists and that such a compilation would 
be useful to adult basic and literacy educators. Investigation of "research" reports also unin- 
tentionally revealed some disturbing discoveries about several key issues related to Section 
353 projects and their dissemination. 

It seemed appropriate that a survey of such projects be undertaken in Pennsylvania since 
this state appears to have the most research projects, if not the most 353 projects of all types, 
on file with the DAEL (Division of Adult Education and Literacy, U.S. Department of Educa- 
tion) and ERIC. (Iowa has spent more special-projects money on research, and its research 
projects are more comprehensive than Pennsylvania's.) However, the actual reason Research 
Distilled was conducted in Pennsylvania was that the idea was conceived here as a result of 
some discussion at a meeting of the FOCUS on Literacy panel. That FY 1993 project was one of 
a series designed to disseminate information on exemplary special projects in Pennsylvania. 
One panel member commented that it might be useful to collect and summarize Section 353 
research projects separately. Initially, the idea for Research Distilled was to collect and sum- 
marize such projects produced only in Pennsylvania. Later, the concept was broadened to 
include research projects from all states. As such, this should by rights have been a national 
project, but 353 funding is granted on a state-by-state basis. 

Procedures for meeting each objective were as follows: 

Objective 2. Collect reports of research projects conducted under Section 353 special-projects 
grants in Pennsylvania and elsewhere in the United States since at least 1989. 

We began under the assumption that we would simply request a list of all Section 353 
projects from the DAEL, then cull that list for projects classified as "research" and request 
copies of those projects' final reports. However, repeated requests to DAEL for a complete list 
of 353 projects were unproductive, though two staff members told us that such a list was 
available and promised to send it. As we later discovered, the most complete listing DAEL 
possesses is in fact only whatever states submit. Based on several telephone conversations 
with key DAEL personnel and the fact that the only listing we received was the DAEL publi- 
cation, A Guide to Special Demonstration and Teacher Training Projects: Special Answers for 
Special Needs (July 1993), it finally became apparent that no comprehensive listing of all 
projects funded under Section 353 monies currently exists. 

We concluded that if DAEL did not maintain a central list of special projects, then surely 
no one else would. We then proceeded to contact each State Literacy Resource Center directly. 
For about half the states, we had to contact State Adult Education Directors rather than Lit- 
eracy Resource Center Directors because some resource centers were just being formed. We 
received only a few responses from our first mailing (with a letter signed by the former Direc- 
tor, Bureau of Adult Basic and Literacy Education, Pennsylvania Department of Education). 
We then learned secondhand that a discussion at a meeting of State Literacy Resource Direc- 
tors had indicated that our request had been misunderstood to be for copies of all 353 projects. 
We subsequently sent a follow-up postcard to clarify and reinforce our request and received a 
few more responses, by phone and mail. 

Nevertheless, from those two contacts with the State Literacy Resource Centers or State 
Directors of Adult Education (three separate contacts for Florida and Texas), we received re- 
sponses from only 13 states: Alabama, Arizona, Florida, Iowa, Kentucky, Massachusetts, Mon- 
tana, North Carolina, Ohio, Oregon, Pennsylvania, Tennessee, and West Virginia. Several of 
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these states courteously responded as per our requests despite having produced no true re- 
search projects. 

Some of the few states from which we received responses did not have, or were unable to 
locate, central listings of funded 353 projects within their own states. A few states were able to 
supply brief abstracts of only the current or the past two years' projects and were unable to 
access any earlier listings. This dearth of record keeping was perhaps even more disturbing 
than the fact that no central listing existed at the federal level. Of those states that did main- 
tain comprehensive lists of 353 projects, none besides Pennsylvania classified projects accord- 
ing to type, such as demonstration or research. Listings were not in any sort of standard for- 
mat, either from year to year within a state or in comparison to other states. 

Florida's projects were exceptionally easy to access. Tucked into the booklet of current- 
year project abstracts was a reader service card on which projects could be requested based on 
assigned numbers. This project utilized that card to request final reports and received two as a 
result (though only one was appropriate for review). 

Of those states which did not respond, we presumed the following reasons: 1] our corre- 
spondence never reached the appropriate person, 2) the state literacy resource center was not 
yet in place, 3) the state had no record of 353 projects, or 4) the state had no research projects 
conducted under Section 353 and so ignored our request. 

Based on the listings received from the states, we requested copies of final reports of all 
projects that appeared to have a research orientation based on the abstracts. Of those received, 
all projects which met our criteria (see below) were reviewed and profiled in the Research 
Distilled publication. 

In summary, we located research projects conducted under Section 353 of the Adult Edu- 
cation Act through the following means: 1) various publications produced by AdvancE, 
Pennsylvania's state literacy resource center; 2) several ERIC listings or other references to 
ERIC document numbers; 3) Special Answers for Special Needs: A Guide to Available 353 
Resources (July 1993), from the U.S. Department of Education, Division of Adult Education 
and Literacy (DAEL) Clearinghouse; 4) at least two contacts to every state literacy resource 
center; and 5) correspondence directly to individual contracting agencies. We maximized use 
of our own State Literacy Resource Center, as Cheryl Harmon of AdvancE fulfilled our numer- 
ous, scattered requests for original documents and ERIC microfiche. Overall, our search was 
conscientious but far from systematic. 

We included additional projects funded under Section 383 (evaluation projects funded 
directly by the U.S. Department of Education through the Research Information Network), on 
the basis of information published in the DAEL Clearinghouse's Resource Update, Although 
that publication told little about each project (only name and contractor, sometimes not even 
the date or funding source), it did provide a separate classification for research projects. We 
located several Section 353 research projects in Resource Update, however, that were not 
classified as such. Some projects are classified incorrectly altogether. As an example, the "Docu- 
ment Type" for a project produced by New Educational Projects was incorrectly listed as "Class- 
room Material." In light of such misclassifications, we pored through lists of all project de- 
scriptions to try to discern from brief abstracts and sometimes from only titles whether a 
project was in any way research oriented. This subjective search may have overlook :d some 
projects that should have been included in our compilation. 

We also found several listings of research projects which, even though contained in 1993 
listings, were too old for this project — as far back as 1983 — so did not pursue them. (One 
exception was "Strategies for Success: A Study in ABE Student Retention," which we in- 
cluded because of its correlations to several other later retention studies.) 

Also, some final reports from 1989-93 were already out of print or reported through direct 
contact with their sources to be "unavailable." Conversely, some FY1993 projects were too 
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recent to have yet entered the document stream. 

After obtaining as many final reports as possible, we began reading. Some of the reports 
were clearly formatted according to standard research-reporting protocol. In some cases, even 
some well-written reports, it was impossible to discern whether a study had been conducted 
on solid ground, that is whether procedures were clearly planned and carried out, even whether 
the premise was sound. However, from some reports, these factors were obvious, and in sev- 
eral, the report writers were candid in pointing out the study's weaknesses. 

Objective 2. Summarize the purpose, methodology employed, and findings of each relevant 
research project, in a brief, standard format. 

Objective 3. Attempt to contact original researchers to learn what, if any, follow-up or subse- 
quent developments have taken place since their projects were completed. 

Objective 4. Cross-reference related research projects to identify similarities, correlations, 
and/or contradictions. 

From the lists and reports which we studied, we observed that most 353 projects were 
funded as demonstration (or pilot) projects delivering direct student services, or as staff-devel- 
opment projects, such as training workshops. Many projects produced curriculum and staff- 
development materials, as part of a demonstration or not. Only a small number of 353 projects 
had a true experimental design. Some, even if they appeared to have been designed as research 
projects, did not follow standard research procedure; sometimes, even some of the project 
objectives strayed from the central purpose. 

We applied the following criteria for including projects in the publication of summaries: 

a. The project was funded under the Adult Education Act, Section 353 or Section 383, be- 
tween FY 1989 and FY 1993. 

b. The project's final report could be located and reviewed firsthand. 

c. The project fit the criteria of a research project in some way: a controlled study, collection 
and analysis of data, evaluation based on set criteria, etc. Only firsthand research projects 
were included, not literature surveys. 

d. The project's findings were deemed worthy of reporting for any reason: they were exem- 
plary, had broad-based applicability, or had a valid premise not fully realized which other 
researchers could refine and develop in related future studies. 

The format for writing reviews of the reports was as follows: Title, Project Number (if 
there was one), Date (fiscal year), State, Contractor, Purpose, Procedure (summarized), Sum- 
mary of Findings, Comments (subjective), Correlations/Cross-References (to other projects or 
studies, to best of our knowledge with limited research), Contact Information, and For Copy of 
Full Report (where to obtain). 

After reviewing final reports, we wrote a letter requesting follow-up information and 
updated contact information, whenever feasible, to the contact persons listed on the reports. 
That we received only tworesponses to these letters suggests that some contact people may no 
longer be at the institutions listed; however, none of our letters was returned by the U.S. 
Postal Service as undeliverable. Because we received so little usable information in the first 
wave of follow-up letters, which was a time-consuming procedure, we did not pursue the 
r emainder as planned. 

Objective 5. Produce 500 copies of a 32-page booklet of research-project summaries, for dis- 
tribution through [Pennsylvania's State Literacy Resource Centers] to interested adult edu- 
cators. 
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Considerable effort went into designing, formatting, and producing the project's 32-page 
booklet of summaries. A new price quotation for the printing job allowed us to print 600 
copies instead of 500, and to have a two-color cover. As is the policy of New Educational 
Projects, Inc., the booklet was printed on recycled paper. 

Copies were then delivered to AdvancE in Harrisburg and disseminated to the Western 
Pennsylvania Literacy Resource Center, nine regional staff development centers throughout 
the state, and individuals requesting copies. 

Availability of the publication was publicized in What's the Buzzi, Pennsylvania's Adult 
Basic and Literacy Education Newsletter, and through Cheryl Harmon and Chris Kemp, the 
literacy resource center directors. In addition, 114 individual copies were mailed from New 
Educational Projects to each contractor whose project was reviewed and to other key contacts 
with a potential interest in the topic. 



Evaluation 

A survey card was tucked into the first 500 copies of Research Distilled (copy attached). How- 
ever, only eight cards were received as of August 3, 1994. Three were postmarked from Penn- 
sylvania; the remainder were from Illinois, New Jersey, Kentucky, Ohio, and Oklahoma. Re- 
sponses were as follows: 

• Average ratings of the publication on a scale of 0 to 5, with 5 being the most favorable 
rating: 

Organization of material covered: 4.75 (7/8 responses were 5) 

Writing quality of material: 4.875 (7/8 responses were 5) 

Presentation/graphic appeal: 4.625 (5/8 responses were 5; one commented 

"Excellent" beside rating of 5) 
Potential as an aid to literature search: 4.75 (7/8 responses were 5) 
Value as ac'ult education reference source: 4.875 (7/8 responses were 5; one added + to 

rating of 5) 

(NOTE: 4 of the 8 respondents rated every item "5. ") 
Other: 

"Documentation of failure to require dissemination of project results through dissemina- 
tion systems: 5" 

"Like complete contact names/for future reference: 5" 
"Value to me: 5" 

• Respondents' reasons for having obtained a copy of Research Distilled: 
"Reference source and duplications avoidance" 

"Regional Staff Dev. Ctr. (Obtained to share w/others)" 

"Information Update" 

"Personal use" 

"As part of my ERIC work" 

"New Jersey Literacy Resource Center - Trenton" 

"I was sent a copy because one of our projects was listed." 

• Other comments: 
"Good job!" 
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"I would like to see this approach done w/PA's 353's each year." 

"So often, projects seem to 'just be' after completion of funding year. I was excited to be 
critiqued, validated as a pro! Thanks!" 

"Very useful. 'Comments' sections informative." 

"Tana - This publication is great. It is a testimony to the difficulty of locating these 
materials because of lack of requirements to disseminate. I think the write ups on projects are 
excellent + provide very helpful evaluation comments - My only difficulty was in understand- 
ing how you were using the word 'experimental' in the intro. At first I thought it meant you 
were limiting to experimental (quantitative) research. Thanks for sending this to me. Susan 
Imel [Director and Adult Education Specialist, ERIC Clearinghouse on Adult, Career, and 
Vocational Education]" 

"Very useful categories for listing of projects." 

Also, in the consultant's last review of the product, Dr. Zellers wrote to the Project Direc- 
tor: 

"Your final product is very attractive and scholarly. You did a beautiful job of formatting 
the information. I can tell that this project was far more than you originally planned ... I wish 
you well and encouragement in future 353 projects. You have consistently produced the high- 
est quality products in this funding area of adult education and I hope you continue ..." 



Comments on Research Under Section 353 

Because the procedures of this project revealed a number of realizations regarding Section 353 
projects and their dissemination, we herewith provide additional comments, based on the 
research conducted by this project, much of which far exceed the original scope of the project's 
objectives. 

By way of introduction, Section 353 of the Adult Education Act (Public Law 100-297) 
mandates that not less than 15% of federal Adult Education Act funds allotted to a state each 
year be used for Special Demonstration Projects and Teacher Training. Two-thirds of the 15% 
must be used for teacher-training (staff-development) projects. 

This funding area has been controversial since the 1960s when it was initiated, and 
throughout its history in 353's predecessors, the related Sections 309 and 310. There are many 
reasons for the controversy, outlined well in both The Adult Education Act: A Guide to the 
Literature and Funded Projects (Leahy) and Ends or Means: An Overview of the History of the 
Adult Education Act (Rose). Lack of clear guidelines and standards, along with poor dissemi- 
nation, have plagued this integral opportunity to develop adult education in the field. 

The original Section 309 of the Adult Education Act of 1966 was "not for basic research 
but rather for demonstration projects and 'other applied research activities' " (Rose). In fact, 
few of these so-called applied research activities have ever employed any research design what- 
soever in their special projects. 

We did locate a Section 310 research project from FY1 98 1, in which Michael A. Ci ivarella 
at Shippensburg State College (now Shippensburg University) in Pennsylvania developed "A 
Comparison of Community Linkage and Outreach Models for Maximizing Adult Basic Educa- 
tion." That study surveyed a variety of ABE programs to discover, comparatively, how they 
were interacting with the general public, particularly through advisory councils and public- 
awareness activities. The project is not reviewed here because of its early date; it is mentioned 
to illustrate the existence of projects funded under adult education special-projects grants that 
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did employ applied research methodology. 

In fact, experimental design of projects is implicit in the Adult Education Act. The Fed- 
eral Register of October 28, 1991, describes the Adult Education State Plan required for sub- 
mission by each State Education Agency (SEA) applying for an Adult Education Act basic 
grant. Not less than 15% of the grant is to be used for "special experimental demonstration 
projects and teacher training projects." Although each state may establish its own criteria and 
priorities for administering special projects, and two-thirds of the 15% must be reserved for 
training personnel, the word experimental in reference to demonstration projects clearly im- 
plies encouragement of the use of experimental methodology. (The Federal Register provides 
no further detail on how the money is to be spent.) 

Unlike many states, Pennsylvania's 353 Projects Request for Proposal explicitly lists Re- 
search under types of "Special experimental demonstration proposals designed for regional or 
statewide impact": 

"2. Research . Proposals in this area should involve the collection and analysis of adult 
basic education data and subsequent recommendations for action." 

Within this category lies the possibility for conducting research, evaluating or surveying 
a situation or population related to adult basic and literacy education, or designing a demon- 
stration project using established research design principles, that is to test a hypothesis. Stan- 
dard procedure for accomplishing this in an academic setting would be to compare a control 
group and an experimental group, or to produce some form of new data. These outcomes have 
rarely occurred in projects of any nature conducted with Section 353 funding. 

Also falling within Section 353 is Teacher Training, and within that area falls "action 
research." This rather new approach provides a good way to answer practical questions in a 
real situation. 

Through its review of K-12 literature, the Study of ABE/ESL Instructor Training Ap- 
proaches identified action research, or practitioner-as-researcher, as "an effective staff devel- 
opment technique. Through action research, teachers identify the questions that interest them 
and plan for and conduct systematic inquiry in their own teaching environments as they work 
with their students. In conducting this research, they keep careful records on the progress of 
learners. The information and insights gathered through this process are used by teacher- 
researchers to improve their own practice and/or to share with others." 

The Statewide Adult Basic Education System (SABES) in Massachusetts follows a sys- 
tematic six-step process for organizing action research projects: 

1. Learn how to look at the classroom. This might include keeping a journal or discussing 
issues with other practitioners to identify researchable problems and exploring possible 
interventions. 

2. Form a research question. The question should be interesting, focused, and finite enough 
to be answered within the constraints of the practitioner's time and resources. 

3. Create a research design. The research must utilize quantitative and/or qualitative meth- 
ods. 

4. Collect data. Collection methods include keeping records, surveying published research, 
collecting students' writing, and administering tests. 

5. Analyze data. This could be in the form of a summary, graph, or chart. 

6. Share results. This might be informally, with students or other teachers, or formally, in a 
report or a presentation at staff-development workshops. 

Or, as summarized by Quigley (1994), "Action research ...has four distinct steps: l)Plan, 
2) Act, 3) Observe, 4) Reflect." This cycle can then be repeated "to test new insights and 
systematically observe and document how each new insight affects our practice." 

A similar procedure may be followed by a "study circle," in which a group of teachers 
collaborate to conduct the action research and meet regularly to share discoveries and tech- 



ERLC 



11 



RESEARCH DISTILLED / FINAL REPORT 



11 



niques and discuss ways to improve both their teaching practice and their staff-development 
program. 

All of these related procedures employ an experimental design, even if the research is 
qualitative in nature. 

Research Distilled located several projects conducted by the Virginia Adult Educators 
Research Network, which is funded under Section 353 to provide a system for coordinating 
action research. The Network appears to be a model of adult education action research in that, 
though cooperative, coordinated efforts, research findings can be applied more generally than 
would be possible in isolated research efforts involving sample sizes too small to be broadly 
valid. We were impressed with the work of this Network, through telephone conversations 
with its director, Suzanne Cockley, and through reading the materials it produces, notably a 
guide to action research that would be useful in virtually any adult education environment. 

In Pennsylvania, small grants to conduct teacher action research are currently available 
through the state's nine regional staff-development centers. In addition, an outlet for publish- 
ing findings exists in The PAACE Journal of Lifelong Learning, the publication of the Pennsyl- 
vania Association for Adult Continuing Education. Teachers are encouraged to contact the 
coordinator or their regional center for further information on action research grants. 

What do those who have conducted research under Section 353 have to say regarding the 
use of Section 353 funding for research projects? Our follow-up letter to project directors asked 
for comments. Herewith selected responses: 

From Dr. Meryl K. Lazar, University of Pittsburgh: 

"The limit of one funding year puts serious constraints on our ability to make any kind of 
impact. It takes most of the year to develop the instrumentation we hope to study and then we 
are left with little time for implementation and follow-up. Too often it takes instructors the 
entire year to feel comfortable with us and the procedures we are asking them to integrate into 
their classroom routines and then the grant ends. Therefore, longterm effects are probably 
negligible. A two-year funding cycle would be great or some kind of renewable option in which 
funding could be renegotiated for a 3-5 year period. We could really accomplish something 
tl m! 

"Another area which we would like to see addressed is for 353 grant monies to be used for 
more pure research. Funding is generally earmarked for [demonstration] projects rather than 
research. To fully understand the types of programming which are effective, some controlled 
studies need to be conducted so that variables of effective instruction can be identified. Once 
we better understand what works, we can then make data-based recommendations on instruc- 
tional programming which will be effective." 

ST" 

From David W. Fluke, Adult Education Linkage Services, Troy, PA: 

"It is certainly feasible and, we feel, appropriate to conduct research projects under Sec- 
tion 353 grants. However, as with so many 353 projects, nearly all of these research studies 
produce no follow-through ... We feel the answer to this is to fund research projects for two 
years (this is permitted under federal Section 353 guidelines) with the initial year being the 
research/data gathering activity and the second year the follow-up, clarification, activity-gen- 
erating year." 
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Conclusions 

The publication produced by this project contains the specific conclusions drawn as a result of 
having reviewed 47 Adult Education A ct research projects. 

In addition, the scope of this project expanded considerably as we realized the difficulty of 
systematically locating final reports of research conducted under Section 353, the special- 
projects section of the Adult Education Act. Further, reading a significant number of 353 final 
reports (having read many others in the past as well) illuminated common flaws in project 
design. Many of 353's problems are under study at state and federal levels, and in some cases 
solutions may be in the works. Based only on the experience and somewhat incidental re- 
search of this project, however, we here relate our observations and conclusions regarding the 
issues of project design and dissemination as of mid- 1994. 

Project Design. Few special demonstration projects were found to be research oriented, 
that is offering a control against which the effectiveness of experimentation can be compared 
and evaluated. This can be viewed as a definitional shortcoming of most "special demonstra- 
tion" projects in actual practice. 

Typical of this flaw was a 1993 Tennessee project which developed an elaborate fiber- 
optic-based interactive telecommunications network for ABE in five counties. The project 
reported gains in reading and math scores of adult students enrolled in the computer-based 
program without any comparison to scores achieved through a traditional delivery system, 
because a control group did not exist. The program may indeed have been successful, but that 
conclusion cannot be definitively drawn in the absence of comparative data. 

Demonstration projecs also often evaluate the innovative curriculathey have produced 
with only post -measurement. For example, a project in New York developed detailed objec- 
tives and curricula and conducted an extensive evaluation of a contextualized vocational edu- 
cation program. However, post-program questions such as "Students exhibit greater knowl- 
edge in the world of work" not only ask for comparisons in the absence of pre-program data 
but also are quite broad and subjective. 

Well-intended demonstration projects such as these that offer no valid quantitative or 
qualitative data were numerous and were not summarized in the Research Distilled publica- 
tion because they did not fit our definition of a research project. Demonstration projects offer- 
ing proof that the effort was successful would be much more useful for everyone's purposes. 

In addition, in our preliminary review of project abstracts, we were struck by fact that so 
many staff workshops funded under Section 353 are presented with objectives such as "To 
train teachers, coordinators and recruiters in strategies that will increase student retention 
and goal attainment." Such a statement assumes that the training is a dissemination of re- 
search-supported strategies. But almost never was this actually the case. Not only is research 
under Section 353 scant, but many contractors are developing staff-development and special- 
demonstration projects with no discernible experimental or research design. It is not enough 
to report a project as "successful" or results as "positive." Projects producing print materials 
probably have less obligation to use a research model. 

The Adult Education Act of 1966 funded experimental and demonstration projects for the 
first time through its Section 309b, the precursor to Section 353. In 1969, General Electric 
Company evaluated the first few years of this new funding and concluded that some useful 
results were indeed emerging. The GE report added, however, that (quoting Leahy) "the projects 
could be more successful and the outcomes improved if solid, better guidelines were estab- 
lished, if assistance and advice were available, especially in the early stages, and if consulta- 
tion were provided for directors who had not had previous experience with such projects. The 
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report also suggested more careful monitoring, training in report writing,, and guidelines for 
dissemination and replication. It recommended that the advantages of implementing the in- 
novations in ongoing programs across the country should be tested and demonstrated." 

Today, some 25 years later, the same conclusions could be drawn with regard to most 
Section 353 projects and to research projects conducted under Section 353 in particular. Much 
work remains to be done to assure higher quality of Section 353 projects. 

A trilogy of documents addressing special-projects evaluation was developed in New York 
state. In A Source Book for Evaluation Special Projects ( 1 988), the authors list types of projects 
conducted under Section 310 (now 353). For demonstration projects, they describe a variety of 
formal evaluation designs; first on the list is "controlled studies using random assignment or 
quasi-experimental designs." Other methods include pre-post outcome measures, follow-up 
investigations, and analysis of data across demonstration sites. 

Later in that report, selected research designs for student impact evaluations are detailed. 
The authors cite the limitations of conducting pre- and post-testing within only one group; 
instead, they recommend the use of the nonequivalent control method, which employs com- 
parison groups of students who are or are not using the project's program design. At the very 
least, performance of a program group can be compared to norms instead of to a comparison 
group, if norms are available. The report also briefly describes how to select students and 
randomly place them into groups to increase the chances of valid results. A variety of model 
evaluation instruments is provided in the appendices. 

The purpose of demonstration projects, says the New York report, is to demonstrate 
whether or not a project is worthy of replication elsewhere in the state or of statewide adop- 
tion. "These projects are of greatest value ... when they include a formal evaluation of their 
effectiveness and efficiency." This determination can be made only through sound project 
design. And at the heart of sound project design are measurable objectives: objectives that 
provide a comparison against which success can be measured. 

Section 353 projects, in Pennsylvania at least, are subjected to strict financial scrutiny, as 
strict as instructional programs are. Budget sheets are meticulously examined by at least three 
offices within the state Department of Education, and are required to comply, in every line 
and column, with rigid guidelines. Audits demand documentation for every purchase, even 
those for as little as one dollar. Frequent financial reports must be filed punctually or funds are 
withheld. Some of these administrative exercises cost inordinate time and stress in the name 
of accountability. Why, then, should not the design and content of our special projects be 
subjected to comparable standards of quality? 

Project Dissemination, Further complicating matters is the uncoordinated dissemina- 
tion system to which 353 projects have had access. 

A 1973 study of adult basic education teacher training under Section 353 (Spear et al.) 
drew a conclusion that could also readily apply to Section 353 today: "Its literature is scat- 
tered; its records imprecise or missing altogether; its costs uncalculated; its students and teach- 
ers uncounted and unknown; its objectives obscure; and its organization adrift." 

In 1979 the National Conference of Adult Education State Directors examined the prob- 
lem of special-projects reporting and dissemination. This group's conclusions (again from 
Leahy) were: 

L Some reports, occasionally excellent ones much needed by the profession, are not reach- 
ing the dissemination systems. 

2. Most reports are not properly designed to accommodate the reproduction requirements 
for dissemination and information retrieval systems; hence, their utility is limited. 

3. Many reports are not reaching the practitioners. Significant reports in some cases are not 
disseminated within a state although the State Education Agency is the grantor. Conse- 
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quently, useful ideas lose their adoptive impact because users and decision makers are 
not informed. 

4. Many [special-projects] reports are incomplete and this diminishes their usefulness. 

Indeed, evaluation, outcomes, and transferability were not even required components of 
projects funded under Section 309/310. New projects seeking to replicate or validate earlier 
projects need to be aware of what others have done. In our search we noted many similar 
projects done in several states, suggesting that grantees either did not (or could not) search the 
literature (as is requested in funding guidelines) or could have adapted another project rather 
than duplicate efforts by pursuing the same basic objective from the ground up. 

An ongoing examination of special projects in Pennsylvania, called FOCUS on . . . (various 
categories such as curriculum, literacy, program management, etc.), conducted by Sherry Roy:e 
and funded under Section 310/353, began in 1985. The FOCUS projects, which continue even 
now, assemble a panel of adult education practitioners to evaluate all 353 projects produced in 
Pennsylvania. Bulletin are then produced describing the "exemplary" projects. This effort has 
gone a long way toward dissemination of successful 353 projects in this one state; however, by 
virtue of FOCUS's emphasis on exemplary projects, any funded project that didn't make the 
cut was likely to languish in obscurity nonetheless. 

On the other hand, despite its efforts to the contrary, the databases of special-project 
reports maintained by ERIC and U.S. Department of Education's Division of Adult Education 
and Literacy (DAEL) Clearinghouse represent neither a complete nor an exclusively "exem- 
plary" listing of projects. In reality, the basic criterion for inclusion is that an individual con- 
tractor or a state made the effort to submit the project. Though DAEL claims that: "Based on 
established criteria, innovative and potentially adaptable projects are selected" and we specu- 
late that some screening often takes place at a lower level (such as within a state), the task of 
judging project quality is limited by the actual pool of projects submitted. 

Then, too, in any dissemination effort which seeks to publicize "exemplary" work, re- 
search projects often have not risen to the top of the heap, for various reasons. Often, as Re- 
search Distilled illustrates, a research project is indeed not exemplary: its methods, even its 
premise, may be flawed, or its outcomes may be incomplete or inconclusive. On the other 
hand, the best-designed research project simply may not be as attention-getting as its demon- 
stration-project competition. The result is that many research projects tend not to be dissemi- 
nated — this in the face of poor dissemination in general. 

Moreover, many 353 projects were never duly recorded as ever having occurred. A Febru- 
ary 26, 1992, memo to State Directors of Adult Education from Dr. Joan Seamon, then DAEL 
Director, expressed the U.S. Department of Education's desire to see all Section 353 project 
summaries: "Please send one copy of the final report and any related products to use within 90 
days after the close of the project. We are presently establishing a database of 353 projects that 
will be an invaluable resource in improving the dissemination of information about special 
projects ... Our publication, Special Answers for Special Needs: A Guide to Available 353 
Resources, is a selected listing of projects based on your reports that are reviewed by special- 
ists on our staff." 

Seamon's request was fundamental but it was not a mandate, and so apparently, at the 
time of publication of Special Answers, some states still were not submitting any Section 353 
project reports while others were not submitting all of their 353 project reports. Many states 
still apparently were not submitting even full lists of funded 353 projects to the DAEL, not to 
mention abstracts, summaries, or final reports. 

Evidence of this discrepancy is an analysis of the contents of the 1993 edition of the 
DAEUs A Guide to Special Demonstration and Teacher Training Projects: Special Answers 
for Special Needs, which lists Section 353 projects covering four or five years. If indeed that 
publication is a compilation of exemplary projects drawn from a complete collection of final 
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reports, then the following statistical analysis is invalid. (Indeed, several of the lists this project 
received from states contained projects not included in Special Answers.) If Special Answers is 
in fact a list of only and all the projects submitted to DAEL, then our observations bear out 
significant omissions. Of the 207 projects summarized in that publication, 59, or 28%, are 
from Pennsylvania. This is presumably due to the fact that Pennsylvania tends to fund numer- 
ous smaller projects rather than a few larger projects, as many other states do. However, this 
disproportionate number of projects from one state, along with the fact that only 28 states and 
the District of Columbia are represented in the publication, implies that not ail states are 
submitting their 353 projects to the DAEL. (Three listed projects indicated no state of origin.) 
Moreover, the fact that several states are represented by only one or two projects suggests that 
some states either are not submitting all their projects, are conducting very few projects, or are 
producing very few exemplary projects. In any case, weaknesses are apparent. 

Further, document retrieval sources are inconsistent. Copies of 353 projects listed in Spe- 
cial Answers are available from the DAEL Clearinghouse, ERIC, ERIC Document Reproduc- 
tion Service, or the original producers themselves. Costs range from no charge to $50 and 
more. Not every title represented in Special Answers is available from DAEL. Two titles re- 
quested from DAEL just five months after release of Special Answers were already unavail- 
able. 

Weaknesses also appeared as we attempted to retrieve 353 documents from individual 
states. Pennsylvania's literacy resource system showed itself to be a model of 353 storage and 
retrieval efficiency. Virtually any project report ever produced in this state is available on loan. 
The few that are not are those that were never microfiched and whose limited copies were 
never returned by borrowers. On the other end of the spectrum, many states, even those that 
do keep a central listing of 353 projects, do not maintain a library of their own state's final 
reports, though most states appear to be in the process of initiating one. This change comes as 
a result of the National Literacy Act of 1991, which amended the Adult Education Act to 
include the State Literacy Resource Center Program, a mandate for the operation of a literacy 
clearinghouse for each state (with some states combined into one regional center). 

Also, under Section 384 of the Adult Education Act, the National Institute for Literacy 
(NIFL) was formed as an. interagency agreement among the Secretaries of Education, Health 
and Human Services, and Labor. The NIFL is designed to support basic and applied research; 
program assistance, technical assistance, and training; policy analysis and evaluation; dis- 
semination of information about best practices of instruction; and assistance to federal agen- 
cies in implementing the Adult Education Act and finding ways to achieve uniform reporting 
requirements, develop performance measures, and develop standards of program effective- 
ness. The NIFL is also charged with serving, through various communication methods, as a 
link among State Literacy Resource Centers. 

Although at this writing many of the new State Literacy Resource Centers are just getting 
started and/or are grossly underfunded (due in part to the required state match; e.g., a reported 
paltry $8,000 for the entire FY1994 in Montana), the new push for better coordination and 
dissemination of resources should eventually produce better documentation of 353 projects. 

In the meantime, researchers have no choice but to gather information on Section 353 
projects from disparate sources, never knowing definitively if what has been collected is com- 
plete. Certainly, it is not, but it may also be presumed that any 353 research project that has 
not been located through the exhaustive methods employed by the Research Distilled project 
is simply not currently accessible or available. 
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Recommendations 

Having perhaps presumptuously far extended the scope of this project, we nevertheless offer 
the following recommendations for Section 353: 

1. All Adult Education Act research and demonstration projects, including basic data col- 
lection and analysis, should be required to follow experimental methodology, even if at a 
simplified level. This would help produce true "best practices" and findings that could be 
validated or replicated elsewhere. Evaluation criteria should be integrated in state- or nation- 
wide special-projects guidelines. Better project design, reporting, and dissemination in general 
would facilitate replication and further validation by other programs with similar needs. 

2. Final reports of Section 353 projects should follow a standard format. In Pennsylvania, 
project producers are given a basic set of requirements and most contractors follow it to vary- 
ing degrees; however, those who do not supply even the basic requirements are not asked to 
resubmit their reports. The result of a lack of final report standards — in any state — is that 
reports may: a) result in projects' being reviewed unfavorably or as less than exemplary, so that 
they never show up at DAEL or ERIC, or even in state bulletins; b) be difficult to read and draw 
conclusions from; c) conceal a flawed project design; d) fail to fully report on the project con- 
ducted; or e) provide inadequate accountability due to weak evaluation components. 

3. Dissemination of Section 353 projects needs to be systematized and broadened. The U.S. 
Department of Education should require abstracts of all 353 projects as a part of fulfilling 
Adult Education State Plans, and all submissions should be classified according the categories 
established by the DAEL Clearinghouse. A copy of every final report should go directly to 
DAEL. At the very least, a list of each state's Section 353 projects would provide DAEL the 
opportunity to request a copy of any report it is particularly interested in seeing. Having actual 
final reports, each on microfiche as well, would ensure that no projects ever disappear into 
oblivion in case anything happens to them at the state level (though states should institute 
similar precautions). In addition, every granted proposal should document awareness of re- 
lated projects completed in other states — information available only through a central listing. 
In addition, there needs to be a way to easily retrieve u.iy funded 353 project's final report, 
preferably from one central source. Exemplary projects could be forwarded to ERIC for distri- 
bution beyond the Adult Education Act community. Within each state, too, methods for 
storaging and retrieving 353 project documents need to be improved. 

4. A special-projects section of the Adult Education Act should be continued, refined, and 
even expanded, as it holds great potential for development of the adult education field if its 
projects are designed, funded, carried out, and disseminated properly. 
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